Algorithms for the Big Screen

When Pauline Leininger goes to the movies, she tries not to focus on the technical effects. “A good movie should make you feel so deeply involved in the story that you don’t give any thought to how it was made.” Yet she could easily tell whether images were filmed in real life, digitally altered, or generated entirely artificially. That’s because as a research associate at the University of Television and Film Munich (HFF), Pauline Leininger researches the way artificial intelligence (AI) is used in film production.

The path that led her into AI research began at LMU, where she completed a bachelor’s degree in media informatics, specializing in human-computer interaction. Early on in her studies, she explored the question of how digital technologies can support creative processes. During her master’s studies at the Institute of Media Informatics, she investigated how AI-generated environments can be used for virtual film productions and developed corresponding digital tools. She then completed her master’s thesis at HFF and submitted it to LMU. Her work received multiple awards, including the AI-HUB@LMU Prize and the ARD/ZDF “Women + Media Technology” sponsorship award (German only).

Today, she works at the Institute of AI in Media Production at HFF. She already knew Professor Silvia Rothe, who holds the chair, from the time they spent working together at LMU. Back then, Rothe was completing her doctorate at the Department of Informatics and continues to supervise master’s theses in the Media Informatics and Human-Computer Interaction research groups. “For me, the link to this topic is perfect,” says Leininger. “That’s because I have a great love of film, but I’m also really interested in computer science and in how new AI models can be incorporated into cinematic workflows in practical ways.”

Dragons, explosions, winter landscapes

Her research focuses primarily on visual AI, which means methods that are capable of generating or modifying images, backgrounds, or three-dimensional scenes. In film production, techniques like these are particularly important for creating visual effects. “Visual effects can take many different forms,” says Pauline Leininger. “They may create explosions in the background, a dragon flying into the frame, or a summer landscape that’s transformed into a winter scene.”

Many of these effects are still created with great human effort. Leininger says that AI opens up new possibilities for supporting these processes. Her own research is also less about automatically generating entire scenes and more about exploring tools that can assist filmmakers during specific stages of the production process.

For an artificially generated scene to appear realistic, certain conditions need to be met. “Objects and environments need to remain consistent,” says Leininger. “For example, a character can’t be wearing a particular jacket in one shot and then suddenly something completely different in the next. Likewise, if the lighting or movements don’t reflect the laws of physics, the audience will spot this immediately.” Human expertise is more important than many people realize. “AI can generate images, but it only ever does what it’s prompted to do,” explains Leininger. “So whether a scene actually works still depends very much on the filmmakers’ understanding of cinematography, acting, and narrative structure.”

Producing fantasy and science fiction at lower cost

The use of AI is currently a topic of intense discussion among filmmakers. “Some see it as a source of new creative possibilities, while others fear the loss of traditional ways of working,” she says. “There’s a great deal of skepticism, sometimes accompanied by a range of concerns.” At the same time, students at HFF frequently turn to Leininger’s team for help with any issues they encounter in production. Their requests range from simple tasks to more complex effects. “It might involve designing a poster that will go on to appear in the background of a scene,” she explains. In other cases, the students want to digitally alter an entire landscape or create a background that would be too expensive to make using traditional methods.

Leininger believes that AI could help to realize ambitious visual ideas for smaller and independent filmmakers, who have often previously been limited by their budget, access to technology, or human resources. “Genres such as fantasy or science fiction, which have long been dominated largely by the big studios, could also become feasible for smaller-scale productions.”

Leininger’s master’s thesis has already led to a new research project. Together with a colleague, she is investigating how real locations can be digitally reconstructed to create a virtual design of them before any filming begins. “Many planning processes still rely on two-dimensional images,” says Leininger. “We’re interested in finding out whether actually being able to walk through a space makes a difference.” The project is based on an AI-supported technology called Gaussian Splatting, which can generate detailed three-dimensional models from photographs of real environments. This allows a filming location to be digitally captured and then made accessible as a virtual space.

Virtual props for production designers

Within this environment, furniture, props, or decorations can be added to give filmmakers an initial impression of the setting. Production designers can experiment with how a space looks, for example, by placing a particular cabinet in a corner or changing the color of a wall – without having to purchase a real cabinet or actually paint the wall. Leininger believes that these kinds of virtual models could also facilitate collaboration within film teams. “If directors, producers, and production designers can walk through a digital set together, this can save time, materials, and costs.”

Beyond her research, Leininger has maintained a strong connection to analog forms of creative work. “I draw, paint, and read a lot, and I have an annual cinema pass,” she says. She also recently added a 3D printer to her desk at home, and she uses it to design little miniatures. She is particularly fascinated by what are known as “book nooks.” “These are small dioramas that resemble miniature streetscapes and are inserted between books on a shelf,” she explains. “I print some of the parts myself, assemble them by hand, and then finally I paint everything. It’s a very tactile process.”

She also does not believe that analog methods in media production will be replaced anytime soon. “New technologies often open up additional possibilities,” she says. “But they don’t automatically replace what came before.” Just as computer animation continues to coexist with traditional stop-motion films, AI-supported productions could exist alongside conventionally produced films. “Originality, emotions, and creative craftsmanship will always remain essential,” Leininger is certain. “No technology can replace that.”